In this paper we introduce the invex programming problem in Hilbert space. The requisite theory has been established to characterize the solution of such class of problems. | Yugoslav Journal of Operations Research 25 (2015), Number 3, 379–385 DOI: ON INVEX PROGRAMMING PROBLEM IN HILBERT SPACES Sandip CHATTERJEE Department of Mathematics,Heritage Institute of Technology, Kolkata-700107, West Bengal,India functionals@ Rathindranath MUKHERJEE Department of Mathematics, The University of Burdwan, Bardhaman-713104, West Bengal,India rnm bu math@ Received: October 2014 / Accepted: March 2015 Abstract: In this paper we introduce the invex programming problem in Hilbert space. The requisite theory has been established to characterize the solution of such class of problems. Keywords: Invexity, Compactness, Weak topology, Frechet derivative. MSC: 26B25, 26A51, 49J50, 49J52. 1. INTRODUCTION The mathematics of Convex Optimization was discussed by several authors for about a century [2, 3, 4, 5, 9, 10, 15, 17, 23, 24]. In the second half of the last century, various generalizations of convex functions have been introduced [2, 3, 4, 5, 6, 7, 10, 11, 12, 14, 16, 18, 19, 20, 22]. The invex(invariant convex), pseudoinvex and quasiinvex functions were introduced by in 1981 [14]. These functions are extremely significant in optimization theory mainly due to the properties regarding their global optima. For example, a differentiable function is invex iff every stationary point is a global minima[6]. Later in 1986, Craven defined the non-smooth invex functions [11]. For the last few decades generalized monotonicity, duality and optimality conditions in invex optimization theory have been discussed by several authors but mainly in Rn [6, 11, 12, 14, 18, 19, 20]. The basic 380 , / Invex Programming Problem difficulty of genaralizing the theory in infinite dimensional spaces is that, unlike the case in finite dimension, closedness and boundedness of a set does not imply the compactness. However, in reflexive Banach spaces the problem can be alleviated by working with weak .